Rotational Unit of Memory
نویسندگان
چکیده
The concepts of unitary evolution matrices and associative memory have boosted the field of Recurrent Neural Networks (RNN) to state-of-the-art performance in a variety of sequential tasks. However, RNN still have a limited capacity to manipulate long-term memory. To bypass this weakness the most successful applications of RNN use external techniques such as attention mechanisms. In this paper we propose a novel RNN model that unifies the state-of-the-art approaches: Rotational Unit of Memory (RUM). The core of RUM is its rotational operation, which is, naturally, a unitary matrix, providing architectures with the power to learn long-term dependencies by overcoming the vanishing and exploding gradients problem. Moreover, the rotational unit also serves as associative memory. We evaluate our model on synthetic memorization, question answering and language modeling tasks. RUM learns the Copying Memory task completely and improves the state-of-the-art result in the Recall task. RUM’s performance in the bAbI Question Answering task is comparable to that of models with attention mechanism. We also improve the state-of-the-art result to 1.189 bits-per-character (BPC) loss in the Character Level Penn Treebank (PTB) task, which is to signify the applications of RUM to real-world sequential data. The universality of our construction, at the core of RNN, establishes RUM as a promising approach to language modeling, speech recognition and machine translation.
منابع مشابه
CO2 Removal from Air in a Countercurrent Rotating Packed Bed, Experimental Determination of Height of Transfer Unit
Carbon dioxide capture is a key issue in climate change mitigation. For decades the removal of carbon dioxide has been an essential step in many industrial processing operations such as the synthesis of ammonia, natural gas purification, and oil refining. In this study, a rotating packed bed has been designed for absorption of carbon dioxide from an air stream. The rotating packed bed is a comp...
متن کاملLookaside Techniques for Minimum Circuit Memory Translators
This paper demonstrates two improvements in coding techniques that could be used for memory word coding. First, within the fixed structure of a Hamming single-error-correcting, double-errordetecting (SEC/DED) code, an improvement can be obtained in circuit cost and operational speed over more conventional code implementations. Second, the mechanics of error correction in a fault-tolerant comput...
متن کاملDesign of a Multiplier for Similar Base Numbers Without Converting Base Using a Data Oriented Memory
One the challenging in hardware performance is to designing a high speed calculating unit. The higher of calculations speeds in a computer system will be pointed out in terms of performance. As a result, designing a high speed calculating unit is of utmost importance. In this paper, we start design whit this knowledge that one multiplier made of several adder and one divider made of several su...
متن کاملPsychometrics and Validation of the Intensive Care Unit Memory Assessment Tool in the Iranian Population
Background: The intensive care unit memory (ICUM) assessment tool is a practical tool for memory monitoring after the discharge from ICU. Objectives: This psychometric study purported to validate ICUM for a sample population of Iranian patients hospitalized in ICU. Materials & Methods: This research was a descriptive-analytical study that was conducted at Ahvaz University of Medical Scienc...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
- CoRR
دوره abs/1710.09537 شماره
صفحات -
تاریخ انتشار 2017